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I .  INTRODUCTION 

Pound-for-pound,  avionics  is  generally  recognized  as 
the  most  expensive,  complex  and  sophisticated  part  of  an  air¬ 
craft^  In  some  modern  aircraft,  the  avionics  may  cost  as  much 
as  30  percent  of  total  flyaway  cost,  and  avionic  support  costs 
\  for  some  older  aircraft  reportedly  approach  three-fourths  of 

\  total  support  cost  [1] . 1 

\ 

"  — ^Reliability  and  maintainability  are  critical  character¬ 
istics  that  influence  spares  provisioning,  maintenance  resource 
requirements,  system  operational  availability  and,  ultimately, 
investment  and  operating  costs.  Consequently,  much  attention 
has  been  focused  upon  improving  the  reliability  and  maintain¬ 
ability  of  avionics  which,  in  the  past,  have  been  disappoint¬ 
ingly  low.,  Mr.  Norman  R.  Augustine,  a  vice-president  of 
/-  'Martin  Marietta  Aerospace,  who  served  as  Under  Secretary  of  the 
/  Army  and  Assistant  Director  of  Defense  Research  and  Engineering 
|  in  OSD,  has  reflected  this  disappointment  in  one  of  his 
|  published  tongue-in-cheek  ’’laws'’  based,  in  part,  on  an  IDA 
study  done  in  1973.  His  law  Number  XIX  states: 

\ 

If  you  are  willing  to  spend  enough  money  on  an  item  of 

electronics,  you  can  virtually  guarantee  that  it  cannot 
\  be  made  to  work. 

But  perhaps  things  are  not  quite  that  bad. 

--..A  'j  *, 

"The  paper,  I  am  presenting,  co-authored  by  Mr.  Joseph  W. 
Stahl  and  me, Addresses  avionics  reliability  and  maintainability. 
The  analysis  was  performed  last  year  in  partial  response  to  a- 

1  Numbers  in  brackets  are  keyed  to  the  list  of  references  at  the  end  of 
this  paoer. 
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request  of  IDA  by  OASD  (C^I)  to  provide  information  for  use  by 
DSARC  principals  at  the  Full-Scale  Development  milestone.  Since 
the  5000  series  of  DoD  Directives  and  Instructions  emphasize 
analytical  comparisons  of  any  new  system  under  consideration  by 
the  DSARC  with  current,  comparable  systems,1  we  undertook 
analyses  to  determine  if  there  may  be  one  or  more  historical 
relationships  between  field  reliability  and  maintainability  and 
avionic  equipment  characteristics  that  would  assist  in  forecasting 
those  attributes. 

The  remainder  of  this  paper  is  devoted  to  discussion  of 
analyses  and  findings  that: 

-  Suggest  and  exemplify  methods  for  projecting  field 
reliability  and  maintainability  of  avionic  systems, 

-  Show  predictive  relationships  between  field  reli¬ 
ability  and  avionic  equipment  unit  cost,  and 
maintainability  and  unit  cost, 

-  Demonstrate  that  field  reliability  and  maintain¬ 
ability  vary  with  the  level  of  technology  incorpo¬ 
rated  in  avionic  equipment, 

-  Illustrate,  by  example,  why  the  reliability  of 
avionics  in  the  field  invariably  has  been  lower 
than  demonstration  and  qualification  tests  indicate. 

II.  THE  DATA  BASE 

Our  historical  data  base  comprises  the  hardware  unit 
costs  and  failure  and  maintenance  data  for  each  of  25  Air  Force 
avionic  equipments.  The  cost  data  are  taken  from  a  proprietary 
data  base  that  represents  the  100th  unit  cost,  in  FY  1978 
dollars,  of  a  number  of  avionic  systems.  The  failure  and  main¬ 
tenance  data  are  from  the  Air  Force  Logistic  Command's  D056  Log 
reports  for  the  six-month  period  August  1979  through  January 
1980  [2]  . 


Regarding  reliability  and  maintainability,  this  requirement  for  comparisons 
with  current,  comparable  systems  is  contained  in  paragraphs  l6.a.  and  b., 
Enel.  4,  DoDI  5000.2,  March  19,  1980;  paragraphs  D.l.c.(2),  and  2. a. (4)  and 
3*a.(2)  (Enel.  3)  DoDD  5000.39,  January  17,  1980;  and  paragraph  D.7b.,  DoDD 
5000.40,  July  8,  1980. 
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The  cost  data  base  identifies  equipments  as  to  hardware 
technology  levels.  The  two  most  recent  technologies  are  used 
in  these  analyses.  The  older  of  the  two  is  identified  as  Solid 
State  Devices  (SSD)  while  the  newer  equipments  are  identified 
as  Micro-Miniaturized  and/or  Complex  Integrated  Circuit  Devices 
(MM).  They  range  from  low-cost  automatic  direction  finders, 
radar  altimeters  and  TACAN's,  to  high-cost  radars,  computer 
sets  and  ECM  equipment.  Dates  of  manufacture  span  the  period 
1965  to  1978. 

The  failure  and  maintenance  data  are  for  avionics 
included  in  the  cost  data  base  and  installed  in  the  following 
aircraft:  F-4C,  F-4D,  F-4e,  F-4g,  RC-135A,  C-130B,  C-130E, 

F-15A,  F-15C,  C-141A,  C-5A,  F-lxlA  and  F-111E.  Not  every  item 
of  equipment  was  installed  in  every  aircraft  so,  for  the  purpose 
of  this  study,  where  an  item  of  equipment  was  installed  in  more 
than  one  aircraft,  the  totals  of  the  reported  operating  hours,1 
failures,  maintenance  actions,  etc.,  were  computed. 

III.  FIELD  RELIABILITY  VS.  UNIT  COST 

It  Is  generally  accepted  that  reliability  and  unit  cost 
are  functions  of  equipment  complexity.  If  this  is  so,  mean  time 
between  failures  (MTBF)  and  unit  cost  may  be  assumed  to  exhibit 
reasonable  correlation.  It  would  follow  that  since  failures 
generate  maintenance  actions,  relatively  good  correlation  might 
be  expected  between  MTBF  and  MTBMA  (mean  time  between  maintenance 
actions)  and  between  MTBMA  and  unit  cost. 

Table  1  presents  the  data  used  in  this  reliability 
analysis.  The  equipments  are  not  identified  so  as  not  to 
disclose  "privileged  information." 

lTime  in  use  is  presented  in  D056  reports  as  operating  hours.  D056  guide¬ 
lines  [3]  prescribe  that  operating  hours  shall  be  computed  by  multipliers 
(factors)  determined  by  the  item  Managers  and  applied  to  reported  flying 
hours.  The  guidelines  acknowledge,  however,  that  the  ratio  of  operating 
timet o- flying  hours  normally  is  1.00.  D056  report  data  used  in  this  study 

bear  out  the  predominant  use  of  1.00. 
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The  results  of  regression  analysis,1  using  unit  cost  as 
the  Independent  variable  and  MTBF  as  the  dependent  variable, 
are  depicted  in  Figure  1.  Examination  of  the  plot  leads  to 
three  interesting  'observations: 

•  Field  reliability  (MTBF)  varies  inversely  with  a  power 
of  the  unit  cost  for  Air  Force  avionic  equipments  of 
the  same  technology  level.  Reasonably  good  correlation 
is  indicated,  reinforcing  the  hypothesis  that  MTBF  is 

a  function  of  unit  cost. 

•  Two  distinct  curves  reflect  the  two  technology  levels 
represented  in  the  data  sample. 

•  The  newer  micro-miniaturized  technology  generally 
provides  higher  reliability  for  a  given  cost;  roughly 

a  factor  of  five  over  the  older  solid-state  technology. 

A  similar  regression  analysis  was  performed  using  MTBMA 
as  the  dependent  variable.  Results  are  shown  in  Figure  2. 
Notably,  the  three  observations  made  regarding  MTBF  versus  unit 
cost  can  be  made  for  MTBMA  versus  unit  cost.  The  major  differ¬ 
ences  between  the  MTBF  equations  and  the  MTBMA  equations  for  the 
same  technology  level  are  the  linear  coefficients.  The  slopes 
of  the  equations  are  nearly  the  same.  This  suggests  that  for 
a  given  technology  level.  If  one  knows  the  MTBF  or  MTBMA,  one 
can  approximate  the  other  based  on  the  ratios  of  the  linear 
coefficients . 

The  similarity  of  the  MTBF-versus-unit-cost  and  MTBMA- 
versus-unit-oost  relationships,  coupled  with  reasonably  good 
fits  of  the  data  points  to  the  regression  lines,  prompted 
exploration  of  a  direct  relationship  between  MTBMA  and  MTBF. 
Figure  3,  developed  from  data  in  Table  1,  presents  the  results 
of  regression  analysis  of  MTBMA  as  a  function  of  MTBF  for  all 
25  equipments  in  the  data  base.  The  dependence  of  MTBMA  upon 


lThe  appendix  comprises  a  discussion  of  the  regression  analyses  performed 
in  this  study. 
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100th  UWT  COST  M  1000*  OP  FT  1978  DOLLARS 


Figure  1.  FIELD  MTBF  VERSUS  UNIT  COST  FOR  SELECTED 
AVIONIC  EQUIPMENTS 


IDOth  UNIT  COST  IN  1000*  OF  FY  1978  DOLLARS 


Figure  2.  FIELD  MTBMA  VERSUS  UNIT  COST  FOR  SELECTED 
AVIONIC  EQUIPMENTS 


* 


Figure  3,  FIELD  MTBMA  VERSUS  FIELD  MT8F  FOR  SELECTED 
AVIONIC  EQUIPMENTS 
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MTBF  constitutes,  a  priori,  a  aausal  relationship.  The  degree 
of  dependence,  however,  is  conspicuous  in  that  the  strong 
relationship  (r*0.97)  appears  to  hold  regardless  of  the  technol¬ 
ogy  level  inherent  in,  or  function  of,  the  equipments. 

IV.  MAINTAINABILITY  VS.  UNIT  COST 

In  addition  to  MTBF  and  MTBMA  data,  the  DQ56  Log  report 
contains  maintenance  man-hours  for  scheduled,  unscheduled  and 
shop  maintenance  actions  at  the  organizational  and  intermediate 
(i.e.,  "shop”)  levels  of  maintenance.1  This  information  enabled 
us  to  evaluate  the  maintainability  specifications  of  an-  avionic 
system  that  was  to  b<*  the  subject  of  a  DSARC  decision. 

The  maintainability  parameter  stipulated  in  the  System 
Segment  Specification  was  mean  corrective  maintenance  time 
( MCMT ) ;  coincidentally,  at  the  organizational  and  intermediate 
levels  of  maintenance.*  The  MCMT  values  specified  did  not 
include  maintenance  set-up  time,  time  to  gain  access  to  the 
equipment,  nor  time  *o  complete  reinstallation.  The  D056  Log 
report,  however,  does  include  those  increments  of  time,  making 
the  man-hours  shown  in  that  report  more  representative  of  the 
maintenance  effort  that  would  be  required  in  the  field.  We 
excluded  scheduled  man-hours  given  in  the  DO 56  Log  report  from 
our  analyses  because  (a)  the  System  Segment  Specification 
prescribed  that  the  equipment  would  not  be  subject  to  scheduled 
maintenance  and  (b)  man-hours  expended  in  scheduled  maintenance 
are  independent  of  failure  occurrence.* 


; Unscheduled  depot  maintenance  man-hours  are  not  Included  in  the  D056  Log 
Report . 

2 

Although  the  need  for  unscheduled  depot  maintenance  was  recognized  in  the 
Integrated  Logistic  Support  Plan,  MCMT  at  depot  level  was  not  addressed 
in  the  System  Segnent  Specification. 

3 Inexplicably,  programed  (i.e.,  scheduled)  depot-level  man-hours  are  includ¬ 
ed  in  shop  (i.e.,  intermediate-level)  man-hours.  The  AFLC  office  responsible 
for  the  D056  Log  Report  TO  attributed  less  than  five  percent  of  reported 
shop  man-hours  to  programed  depot -level  avionics  (continued  on  next  page) 
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Table  2  contains  the  data  used  for  the  maintainability 
analysis  of  the  same  equipments  used  in  the  reliability 
analyses:  equipment  failures,  maintenance  actions,  unscheduled 
man-hours,  shop  man-hours  and  unit  costs.  This  information 
constituted  the  data  base  to  perform  regression  analysis  using 
the  two  variables,  unit  cost  and  unscheduled-plus-shop-man- 
hours  per  failure.  Figure  4  illustrates  the  results.  The 
correlation  coefficient  and  the  scatter  of  points  about  the 
line  representing  solid  state  devices  indicate  somewhat  question¬ 
able  correlation  between  man-hours  and  unit  cost.  The  equation 
generated  by  the  micro-miniaturized  data  points  yields  such  poor 
correlation  (r<0.25)  that  it  is  not  plotted.  The  scatter  of 
the  micro-miniaturized  data  points  illustrates  the  independence 
of  man-hours  per  failure  and  unit  cost.  In  view  of  these  results, 
the  arithmetic  means  of  man-hours  per  failure  for  both  technologies 
are  shown  in  Figure  4  as  horizontal,  dashed  lines.  They  indicate 
that  the  average  man-hours  per  failure  for  the  micro-miniaturized 
equipments  is  about  twice  that  of  the  older  solid-state  technology 
devices.  This  observation,  coupled  with  the  foregoing  analysis 
of  field  reliability,  suggests  that  although  the  equipments 
typified  by  the  advanced  technology  fail  less  frequently,  they 
require  more  man-hours  per  failure  to  repair. 

Performing  regression  analysis  using  maintenance  man¬ 
hours  per  maintenance  action  results  in  relatively  poor 
correlations  and  relatively  large  dispersions  of  points  about 
the  regression  lines  in  Figure  5.  The  arithmetic  means,  however, 
indicate  that  the  more  advanced  micro-miniaturized  equipments 
consume  about  one-third  more  man-hours  per  maintenance  action 
at  the  organizational  and  intermediate  levels  of  maintenance  as 
do  the  older  solid-state  avionics. 


(cont'd)  maintenance.  "Shop"  man-hours  in  the  D056  report  include  time  spent 
on  bench  checks  in  -which  failures  were  reported  but  not  verified.  The  same 
AFTiC  office  advised  that  such  instances  account  for  about  five  percent  of  total 
report  shop  time.  Although  available  information  did  not  permit  selective 
reductions  in  the  D056  data,  calculations  revealed  that  the  effects  of  relevant 
adjustments  to  the  data  would  be  trivial  in  the  context  of  these  analyses. 


VERSUS  UNIT  COST  FOR  SELECTED  AVIONIC  EQUIPMENTS 


Finally,  correlation  of  maintenance  man-hours  per  operat¬ 
ing  hour  and  unit  cost  were  prepared.  Table  3  presents  the  data 
used  to  develop  the  maintenance  man-hours  per  operating  hour 
for  each  piece  of  equipment.  The  same  man-hours  were  used  as  in 
the  previous  discussion.  The  data  points  and  the  equations  are 
plotted  in  Figure  6.  The  correlation  coefficients  are  much  better 
than  those  of  the  previous  regressions  using  man-hours  per 
failure  or  man-hours  per  maintenance  action.  Note  that  the  curve 
described  by  the  micro-miniaturized  (newer  technology)  points  lies 
below  the  curve  described  by  the  solid-state  devices  (older 
technology)  points.  It  has  already  been  shown  that  although 
the  newer-techno logy  equipments  fail  less  frequently,  they  require 
more  man-hours  per  failure  to  repair.  The  results  of  this 
maintenance-man-hour-per-operat ing-hour  analysis  indicate  that 
the  improvement  in  reliability  of  the  newer-technology  equipments 
more  that  offsets  the  additional  man-hours  needed  to  repair 
failures  at  the  organizational  and  intermediate  le^  *ls  of 
maintenance . 1 

V.  SUMMARY 

The  results  of  these  reliability  and  maintainability 
analyses  can  be  summarized  as  follows: 

•  Field  reliability  (measured  by  MIBF  and  MIBMA)  varies 
inversely  with  unit  cost  for  avionic  equipments  of  the 
same  technology  level. 

•  Given  two  avionic  devices  having  the  same  unit  cost,  one 
typified  by  the  newer  micro-miniaturized/complex  integrated 
circuit  technology,  the  other  by  the  previous  generation 
3olid-state  technology: 

1This  finding  does  not  imply  that  all  maintainability  problems  are  offset. 

The  need  for  additional  maintenance  man-hours  per  failure  could  increase 
aircraft  turnaround  times  or,  alternatively,  increase  maintenance  skill 
requirements. 
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Figure  6. 


UNSCHEDULED  +  SHOP  MAN-HOURS  _ _ 

- ap-EHATIWa1  HOURS -  VERSUS  UNIT  C0ST 

FOR  SELECTED  AVIONIC  EQUIPMENTS 
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-  The  field  reliability  of  the  equipment  characterized  by  the  newer 
technology  would  exceed  that  of  the  older  technology  by  a  factor 
of  five. . . .however, 

-  Given  a  failure,  the  equipment  incorporating  the  newer  technology 
may  consume  twice  as  many  man-hours  to  repair  at  the  organizational 
and  intermediate  levels. 

-  Maintenance  man-hours  expended  per  operating  hour  at  the  organiza¬ 
tional  and  intermediate  levels  on  avionics  typified  by  the  newer 
technology  approximates  40  percent  of  the  man-hours  devoted  to 
avionics  incorporating  the  older  technology. 

Regarding  the  results  of  the  foregoing  reliability 
analyses,  we  fully  expect  that  the  quantitative  values  Ce.g.,  the 
equations  and  correlation  coefficients)  would  change  somewhat 
with  enlargement  and  refinement  of  the  data  base.  We  believe, 
however,  that  the  trends  and  relationships,  as  stated  qualitatively 
in  our  findings,  are  unmistakable. 

The  results  of  our  maintainability  analyses  are  less 
satisfying.  Although  indicative  of  the  organizational  and 
intermediate  levels  of  maintenance,  consideration  of  unscheduled 
depot-level  maintenance  is  needed  to  attain  more  complete  under¬ 
standing. 

VI.  TEST  MTBF  VS.  FIELD  MTBF 

In  the  course  of  our  study  of  the  avionic  system  approaching 
a  DSARC  milestone,  some  DoD  officials  concerned  with  the  system 
frequently  referred  to  an  MTBF  value  given  in  the  System  Segment  . 
Specification,  which  referenced  the  applicable  reliability  test 
MIL  STD,  as  expected  field  MTBF.  In  fact,  that  MIL  STD  [5] 
specifies  that  the  MTBF  value  is  "...for  the  purpose  of  establishing 
accept/reject  criteria  and  shall  not  be  used  for  projection  of 
equipment  MTBF . " 

The  fallacy  u_*ng  the  results  of  demonstration, 
development ,  or  qualification  tests  as  representative  of  expected 
field  reliability  is  exemplified  by  the  Failure  Tree  in  Figure  7. 

The  reliability  demonstrated  for  a  similar  avionic  system  by  the 
PSD  contractor  is  used  as  the  example.  Note  that  the  325-hour 


REPORTED 

(97) 


MTBF  AT 
7,817  HRS 

81 


VERIHED 

(67) 


UNVERIRED  117 


RELEVANT 

(24) 


NON-RELEV  ANT 


Figure  7.  FAILURE  TREE 


MTBF  in  the  column  on  the  right  is  based  on  relevant  failures;1 
and  relevant  failures,  by  contractual  agreement,  were  the  basis 
for  determination  of  the  contractor's  compliance  with  reliability 
specifications.  Actually,  97  failures  were  initially  reported  as 
equipment  failures,  of  which  67  were  verified  as  equipment  failures. 
Of  the  67,  only  24  were  eventually  classified  as  relevant,  yielding 
by  contractual  definition,  an  MTBF  of  325  hours. 

In  deployed  operational  units,  however,  discrimination 
between  relevant  and  non-relevant  failures  is  immaterial.  Each 
perceived  (i.e.,  "reported")  failure  generates  a  maintenance  action 
and  each  verified  failure  frequently  generates  spares  requirements 
and  additional  maintenance  effort.  With  reference  to  the  example 
in  Figure  7,  the  DO 56  Log  report  would  have  shown  a  117-hour 
MTBF;  about  one-third  of  the  325-hour  MTBF  resulting  from 
contractual  definition.  Obviously,  the  indiscriminate  use  of 
the  acronym  "MTBF"  can  be  grossly  misleading,  especially  in 
planning  for  O&S  resources  for  new  avionic  systems. 


testing  was  conducted  consistent  with  MIL-STD-781B,  which  defined  relevant  j 
failures  as  those  attributable  to  the  contractor's  performance  (e.g. ,  design,  j 
workmanship  and  premature  part  failure).  Discrimination  between  relevant  j 
and  non-relevant  failures,  however,  are  subject  to  negotiation.  1 
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I 

REGRESSION  ANALYSIS  METHODOLOGY  1 

1 

j 

■3 

In  simple  linear  correlation  the  correlation  coefficient'  J 

Cr)  is  a  parameter  commonly  applied  to  test  the  '’goodness  of  1 

I 

fit"  of  the  data  points  used  to  generate  a  straight  (regression)  i 

line  by  the  method  of  least  squares.  A  correlation  coefficient  j 

of  1  (i.e,,  perfect  correlation)  would  mean  that  all  points  f 

used  to  describe  the  regression  line  fell  on  the  line,  and  that 
estimates  made  from  the  line  (or  its  equation)  would  be  without 
error.  Although  "goodness  of  fit"  should  not  be  confused  with  = 


causality,  normally  the  closer  r  is  to  1,  the  higher  the  con¬ 
fidence  that  can  be  placed  in  estimates  that  fall  within  the  1 

range  of  the  data.  The  square  of  the  correlation  coefficient,  i 

named  the  index  (or  coefficient)  of  determination,  indicates  \ 

the  proportion  of  variation  in  the  dependent  variable  (Y)  that 
can  be  explained  in  terms  of  the  independent  variable  (X) .  j 

Regression  analyses  were  performed  on  historical  cost,  j 


reliability  and  maintainability  data  to  determine  if  relationships  1 
between  cost  and  the  reliability  and  maintainability  variables  \ 

could  be  developed.  The  regression  analyses  were  performed  on 
the  General  Electric  (GE)  Time  Sharing  System,  using  the  standard  I 
GE  program  CURFT$.  The  program  executes  "...a  regression  analysis  j 


It  was  decided  to  utilize  throughout  this  Paper  the  one 
curve  type  of  the  six  in  the  Q.E.  program  that  yielded  the  best 
fit  most  often.  Outputs  of  the  Q.E.  program  were  examined  and, 
through  ranking  the  values  of  the  correlation  coefficients  for 
the  six  curve  types  for  all  of  the  regressions,  the  power  form 

y. 

(Y«AX  )  was  indicated  as  the  choice.  The  power  form  is  a  non¬ 
linear  equation,  however,  and  the  Q.E.  program  transforms  the 
input  data,  as  necessary,  so  that  the  equation  derived  from  those 
data  is  expressed  in  linear  form  to  enable  linear  correlation. 

To  obviate  the  effect  of  data  transformation,  we  examined  the 
values  of  the  means  and  the  standard  deviations1  of  the  differences 
between  the  actual  values  of  the  dependent  variables  and  the  values 
calculated  for  those  variables  from  the  regression  equations  gen¬ 
erated  by  the  G.E.  program.  The  resulting  values  of  standard 
deviation  were  then  ranked,  as  before,  to  select  the  most  repre¬ 
sentative  curve  type.  Again,  the  power  form  was  indicated  as  the 
choice;  accordingly,  it  was  used  throughout  this  Paper,  The  reader 
should  be  aware  that  the  regression  equations,  correlation 
coefficients,  and  figures  in  the  main  text  reflect  the  data 
transformation  performed  by  the  G.E.  program.  This  fact  does 
not  diminish  the  validity  of  the  analyses. 


Standard  deviations  were  computed  using  the  unbiased  estimator;  that  is, 
the  sum  of  the  squared  deviations  was  divided  by  (n-1)  rather  than  (n) . 


